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Abstract 


Recently, commercial vision sensors hit the mobile market. 
To achieve that, computer vision networks had to be quantized. 
However, this topic was not studied well for Image Signal proces- 
sor (ISP) challenging image restoration tasks, being crucially im- 
portant for hardware implementation, as well as for deployment 
on hardware accelerators, e.g. Neural Processors Units (NPU). 

In this paper, we studied the effect of the quantization of deep 
learning network on image quality. We tried various quantization 
on raw RGBW image demosaicing. Experimental results show 
that 10 bit weight quantization can sustain image quality at the 
similar level with floating-point network. 8 bit quantized network 
shows slight degradation in objective image quality and mild vi- 
sual artifacts. 

Although network weight’s bit-depth can be significantly re- 
duced for computer vision tasks, our finding shows that it is not 
true for raw image restoration tasks: at least 10 bit weights are re- 
quired to provide sufficient image quality. However, we can save 
some memory on feature maps bit-depth. We can conclude that 
network bit depth is critical for raw image restoration. 


Introduction 

Deep neural networks (DNNs) have become the state-of-the- 
art in the computer vision and sequence modeling problems like 
image classification, object detection, speech recognition. How- 
ever, they usually comes with high cost computation and memory 
costs from a huge amount of parameters. For example, Reference 
[1] came up with a deep convolutional neural network consisting 
of 61 million parameters and won the ImageNet competition in 
2012. It is followed by deeper neural networks with even larger 
numbers of parameters, e.g., Reference [2]. So it makes the de- 
ployment on the mobile platforms that have limited power and 
computation resources challenging. 

Parameters of a trained neural network commonly ex- 
hibit high degrees of redundancy [3] which implies an over- 
parametrization of the network. Network compression methods 
implicitly or explicitly aim at the systematic reduction of redun- 
dancy in neural network models while at the same time retain- 
ing a high level of task accuracy [4]. Many compression meth- 
ods perform some form of pruning or quantization. Pruning is 
the removal of irrelevant units (weights, neurons or convolutional 
filters)[5]. Quantization is the reduction of the bit-precision of 
weights, activations or even gradients, which is particularly desir- 
able from a hardware perspective[6] 

Network quantization for vision applications like classifica- 
tion, image segmentation and object detection has drawn great 
attention of researchers [1] [2] [7] [8] [9]. Approaches for low-bit 
quantization of neural networks have been made for these appli- 
cations. There are binary weight networks [10] [11] and ternary 


networks [12] [13] [14]. But owing to requirement of high bit- 
depth and high resolution there are no prior art on quantization 
of image restoration problems like demosaicing, super resolution 
and deblurring, etc. 

In this paper, we studied the effect of the quantization of 
deep learning network on image quality. We tried to test various 
quantization bit-depth that is not supported in the conventional AI 
platform like Tensorflow or Pytorch and AI hardware like NPU, 
DSP or GPU. We tried to find what is optimal bit-depth without 
image quality degradation. We could also find the best suitable 
bit-depth for custom AI hardware. 


Releated works 

In this work we focus mostly on demosaicing and quantiza- 
tion algorithms, so we will brifly review related works. 

Demosaicing of bayer color filter array has been extremely 
studied. [15], [16].There are various conventional approches, 
such as color difference based interpolation [17], [18], frequency 
domain filtering [19], [20], [21], and reconstruction methods [22], 
[23]. But for new other patterns, other effort like hand-crafted al- 
gorithms should be applied to solve it. So there is also universal 
approach [24]. 

Deep learning approaches to demosaicing has been applied 
[25], [26], [27], [28]. Previously, many researches focused on the 
bayer CFA demosaicing, but there are researches on Quad bayer 
pattern and Nona pattern demosaicing also [29], [30]. Deep learn- 
ing methods have better image quality in complex CFA pattern 
demosaicing although they require high computation cost. 

Especially we focus on RGBW CFA and its demosaicing. 
There are conventional algorithms like [33], [34] and deep learn- 
ing approaches like [35]. Here our approach is related to deep 
learning RGBW demosaicing. 

To deploy deep network on mobile platform, quantization is 
needed usually. There are two types of quantization methods. It is 
often desirable to reduce the model size by quantizing weights and 
activations post-training, without the need to re-train/fine-tune the 
model. These methods, commonly referred to as post-training 
quantization, are simple to use and allow for quantization with 
limited data [31].Quantization-aware training simulates quantiza- 
tion during training so that the quantization parameters can be 
learned together with the model using training data [32]. 


Problem statement 

Demosaicing can be done for any CFA pattern, but there are 
image quality degradations when using non-standard CFA pat- 
terns, such as RGBW. To enhance image quality for RGBW CFA 
demosaicing, we aim to use deep neural network as shown in 
Fig. 1. Deplyment on a mobile platform requires quantization 
of network. In this work we studied to find how many bits are 


sufficient for the quantized network without image quality degra- 
dation. 
dstributedattention 


Proposed method 

In this work we developed our own network called De- 
PhaseNet. Its features are multi-level network with multi-phase 
inputs to adopt various phase schemes and correlations, dis- 
tributed attention modules for layer-wise proportional sum and 
frequency loss training scheme. In Fig. 2 PE means phase extrac- 
tor that extracts various combination of phases and subnet consists 
of residual blocks and attention blocks. Fig. 3 shows distributed 
attention module. 

The network architecture shows outperforming image qual- 
ity and artifact reduction compared to previous methods. But it 
comes at cost 1.9MMAC/1 pixel. Deployment of this network on 
device needs quantization. 

In this work we used post training quantization to see the di- 
rect effect of quantization on image quality. There are two quan- 
tizations in the network showed in Fig. 4, one is weight quantiza- 
tion and the other is feature map (activation) quantization. How- 
ever, we did weight quantization only putting activations floating 
point in this study, to see how weight bit depth affects image qual- 
ity. We tested quantization with various bits. 


Figure 1. RGBW demosaicing with deep learning network 
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Figure 2. DePhaseNet 
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Figure 3. Distributed attention module 


Weight distribution is close to zero mean distribution so that 
symmetric quantization that we use is effective in weight quanti- 


zation. 
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Figure 4. Quantization in deep network 
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Figure 5. Symmetric quantization 


Experimental results 

We made experiments by preparing pairs of RGBW CFA 
pattern images and ground truth RGB images. The network was 
trained on MIT dataset and HDR+ Burst Photography Dataset 
[36]. We measured our algorithm on Kodak dataset [37] and real 
RGBW-K (kodak) image. 

In Table. 1 and Fig. 6, objective image quality evaluation 
results on various bits are provided. 10 bit shows the image qual- 
ity close to floating point. And 8 bit shows slight image quality 
degradation, 0.43 dB compared to floating point results. 
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Figure 6. Quantized network output results on Kodak image, PSNR [dB] 


Results for weight quantization in Kodak dataset, PSNR [dB] 


Bit PSNR [dB] 
4 27.079 
5 36.003 
6 38.945 
7 41.467 
8 42.485 
9 42.819 
10 42.889 
11 42.913 
12 42.915 
13 42.916 
14 42.916 
15 42.917 
16 42.917 
float 42.917 


Subjective evaluation of experimental results show that we 
could see more quantization noises are shown in lower bit depth. 

In kodak dataset, difference between 8 bit and 10 bit is 
slightly noticeable and 4 bit is worst in image quality as shown 
in Fig. 7 and Fig. 8. 


Figure 7. Quantized network output results on Kodak image number 20: 
(a) - 4 bit; (b) - 6 bit; (c) - 8 bit; (d) - 10 bit. 


Figure 8. 


Quantized network output results on Kodak image number 23: 
(a) - 4 bit; (b) - 6 bit; (c) - 8 bit; (d) - 10 bit. 


And we made tests on 10 bit real RGBW-K raw images, and 
we could see clear differences as shown in Fig. 9 and Fig. 10. 
From 8 bit to lower bits the quantized network results show var- 


ious noticeable artifacts while we could see perfect image recon- 
struction when it comes to 10 bit. In Fig. 11, 8 bit shows artifacts 
on the letter ’o’ and ’n’. 6 bit and 4 bit show color change as well 
as artifacts. 


Figure 9. Quantized network output results on real RGBW-K raw image 1: 
(a) - 4 bit; (b) - 6 bit; (c) - 8 bit; (d) - 10 bit. 


Figure 10. Quantized network output results on real RGBW-K raw image 
2: (a) - 4 bit; (b) - 6 bit; (c) - 8 bit; (d) - 10 bit. 
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Figure 11. Quantized network output results on real RGBW-K raw image 
3: (a) - 4 bit; (b) - 6 bit; (c) - 8 bit; (d) - 10 bit. 


Conclusion 

In this work, we studied quantization on deep learning net- 
work for RGBW-K demosaicing as a type of image restoration. 
Our research shows that 10 bit is most adequate for quantization 
of demosaicing both in objective quality and subjective quality 
aspect. 

If one should deploy the network on usual AI hardware like 
NPU, DSP or GPU with conventional platform such as Tensorflow 


or Pytorch, only 8 bit and 16 bit are supported so that network 
should be quantized with 16 bit weights for best quality in the 
image restoration applications. In cases some artifacts and sligh 
image quality degradation are allowed or artifact reduction post- 
processing like denoising can be used, 8 bit may be used. 


When it comes to the custom AI hardware or dedicated hard- 


ware, 10 bit is best choice for the weight and will make image 
quality close to floating point. 


Recently some AI hardware platforms support float16, but it 


requires 1.5x area and power than fixed point hardware, so that 
integer quantized hardwares have still benefits. 
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